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Abstract 

Background: Ongoing challenges to healthcare integration point toward the need to move beyond structural and 
process issues. While we know what needs to be done to achieve integrated care, there is little that informs us as 
to how. We need to understand how diverse organizations and professionals develop shared knowledge and 
beliefs - that is, we need to generate knowledge about normative integration. We present a cognitive perspective 
on integration, based on shared mental model theory, that may enhance our understanding and ability to 
measure and influence normative integration. The aim of this paper is to validate and improve the Mental Models 
of Integrated Care (MMIC) Framework, which outlines important knowledge and beliefs whose convergence or 
divergence across stakeholder groups may influence inter-professional and inter-organizational relations. 

Methods: We used a two-stage web-based modified Delphi process to test the MMIC Framework against 
expert opinion using a random sample of participants from Canada's National Symposium on Integrated Care. 
Respondents were asked to rate the framework's clarity, comprehensiveness, usefulness, and importance using 
seven-point ordinal scales. Spaces for open comments were provided. Descriptive statistics were used to 
describe the structured responses, while open comments were coded and categorized using thematic analysis. 
The Kruskall-Wallis test was used to examine cross-group agreement by level of integration experience, current 
workplace, and current role. 

Results: In the first round, 90 individuals responded (52% response rate), representing a wide range of professional 
roles and organization types from across the continuum of care. In the second round, 68 individuals responded 
(75.6% response rate). The quantitative and qualitative feedback from experts was used to revise the framework. 
The re-named "Integration Mindsets Framework" consists of a Strategy Mental Model and a Relationships Mental 
Model, comprising a total of nineteen content areas. 

Conclusions: The Integration Mindsets Framework draws the attention of researchers and practitioners to how 
various stakeholders think about and conceptualize integration. A cognitive approach to understanding and 
measuring normative integration complements dominant cultural approaches and allows for more fine-grained 
analyses. The framework can be used by managers and leaders to facilitate the interpretation, planning, 
implementation, management and evaluation of integration initiatives. 
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culture, Change management 
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Background 

Innovative healthcare delivery models often incorporate 
the concepts of integration and integrated care. By bring- 
ing together multiple professionals, services, and organiza- 
tions, integration efforts aim to replace fragmented care 
with care that is coordinated and patient-centered [1]. 
Despite the promise of improved efficiency, quality of 
care, and patient satisfaction, healthcare systems cap- 
able of delivering integrated care have not developed 
widely [2-4]. 

Over the past two decades, most scholars have focused 
on the structural and process challenges involved in inte- 
grating care. The resulting body of knowledge on integra- 
tion barriers and enablers has informed many positive 
system changes. Progress has also been made in under- 
standing, effecting, and evaluating various types of inte- 
gration, including organizational, functional, service, and 
clinical integration [5]. However, reports continue to high- 
light the seemingly intractable problems inherent in fos- 
tering collaboration and cooperation across professional 
and organizational boundaries [6-9]. These problems 
point toward the need to move beyond the integration 
of organizational structures and processes to under- 
stand the social cognitions that are implicated in inte- 
gration efforts. Normative integration, defined as "an 
ethos of shared values and commitment to coordinating 
work [which] enables trust and collaboration in deliver- 
ing healthcare", may be the key to addressing ongoing 
challenges to integrated care delivery [5]; but we still 
know little about how to achieve and measure norma- 
tive integration. 

One way to explore normative integration is by exam- 
ining organizational and professional cultures, defined as 
the beliefs and behaviour patterns dominant among a 
group of people [10]. For example, at the outset of the 
implementation of an integration initiative one may gauge 
the extent to which there are discrepancies among cul- 
tures that will need to be addressed. In turn, post- 
implementation cultural harmonization may be used as 
evidence of normative integration. Although culture is 
rarely directly measured in studies of integration, cultural 
differences are often offered as explanations for failed or 
suboptimal integration initiatives [8,11,12], Despite this 
popular attribution, there is little evidence on the effect- 
iveness of strategies to change culture, and debates persist 
regarding the feasibility and desirability of merging various 
cultures in healthcare [13,14]. 

An alternative perspective is to view integration through 
the lens of individual and shared mental models. Mental 
models are psychological representations, consisting of 
knowledge and beliefs, which enable interpretation and 
action in a specific domain [15]. Mental models are devel- 
oped over time through experience, direct communication 
and interaction with others, and vicarious learning [16]. 



When multiple individuals develop a common psycho- 
logical structure for understanding their environment, this 
is referred to as a Shared Mental Model (SMM) [15,17]. 
Mental models inform perception and behavior, and shape 
culture in a co-evolutionary process [10,18,19]. Like cul- 
tural congruence, mental model similarity may also be 
used as an indicator of readiness to integrate, or post- 
implementation similarity as a sign of successful inte- 
gration. While both constructs, culture and SMMs, help 
to describe and explain normative integration, a mental 
models perspective may allow for more fine-grained 
analyses of inter-organizational and inter-professional 
relations [20]. Mental models may also be more amen- 
able to change than cultures and SMMs may be a means 
for bridging diverse cultures [19,21]. 

Drawing from an interdisciplinary literature review and 
SMM theory, Evans and Baker [20] developed a frame- 
work of mental model content, antecedents, and out- 
comes specific to integration initiatives in the healthcare 
sector. Their framework outlines important knowledge 
and beliefs whose convergence or divergence across stake- 
holder groups may influence inter-professional and inter- 
organizational relations. This study builds on that work, 
and tests and improves the Mental Models of Integrated 
Care (MMIC) Framework through an iterative process 
using expert opinion. An expert-validated framework will 
support the application of a cognitive perspective in future 
research and practice on healthcare integration. 

Theoretical framework 

In this study, we draw from Shared Mental Models 
(SMMs) theory, which posits that team effectiveness is 
maximized when members of a team have a shared under- 
standing of their tasks and roles [15]. SMMs fall into three 
broad categories: task-related (goals and performance 
requirements), team-related (interpersonal interaction 
requirements and skills of team members), and beliefs 
(preferences or expectations) [15]. SMMs in these con- 
tent areas improve team and organizational perform- 
ance by facilitating coordinated action and adaptation 
under changing conditions [15,22-24]. SMMs allow indi- 
viduals to develop common views of what is happening, 
what is likely to happen next, and why it is happening, 
and thus guides their behaviours in ways that are consist- 
ent and coordinated with each other in the completion of 
interdependent tasks [15]. Although diversity in perspec- 
tives has been linked to high decision quality and im- 
proved team performance, this is likely due to the positive 
effects of dissent at the strategy formulation or decision- 
making stage; at the point of action or implementation, 
however, SMMs positively impact performance [25]. That 
being said, identical knowledge and beliefs are not ne- 
cessary; convergence around a broad frame of mental 
models provides the common meaning needed for action, 
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regardless of differing views on specific issues [26]. Thus, 
we define "shared" as similar and overlapping, not identi- 
cal, mental models among a group of individuals. 

Research in strategic management, stakeholder manage- 
ment, change management, systems change, and team, 
organizational and network performance also emphasize 
the importance of developing common aims and shared 
understandings e.g., [27-29]. Although SMMs are group- 
level phenomena, the extent to which individual perspec- 
tives and behaviours are consistent (or not) can impact 
performance beyond the group level [28,30,31]. For ex- 
ample, in a study examining healthcare personnels mental 
models of the organizational implementation of clinical 
practice guidelines, personnel in high-performing facilities 
exhibited SMMs, while those in lower performing facilities 
did not [30]. SMMs help to shape broader constructs such 
as inter-organizational macro- cultures, industry mindsets, 
and institutional logics, which also suggests that SMMs 
can manifest at the systems level [32-34]. Hence, SMMs 
may facilitate the implementation of complex system- 
level change involving the inputs of a multitude of di- 
verse actors. Systems integration, which aims to link 
various sectors, organizations and professionals across 
the continuum of care, is a prime example of such a 
change. For a more thorough justification of the appli- 
cation of SMMs theory to healthcare integration and a 
discussion of strengths and limitations, please refer to 
Evans and Baker [20]. 

Applied to integration, the three mental model categor- 
ies are integration-task, system-role, and integration-belief 
[20]. An integration-task mental model encompasses the 
purpose and approach to integration and includes the 
following contents: services to be integrated; external cus- 
tomers; goals; long-term vision; and processes. A system- 
role mental model refers to ones understanding of the 
system and its components, and includes the following 
contents: knowledge, skills and abilities of participating 
professionals and organizations; role clarity; role inter- 
dependence; role contribution; and interaction patterns. 
Finally, an integration-belief mental model includes rele- 
vant preferences and expectations as well as meanings, as- 
sumptions, and interpretations of key issues. This initial 
framework was derived from a review of previous theory 
and research [20] . In this paper, we examine the extent to 
which the Mental Models of Integrated Care (MMIC) 
Framework, and its dimensions and contents, are relevant, 
acceptable and useful to those with expertise and experi- 
ence in integration efforts, and we revise the framework 
based on this assessment. 

Methods 

We evaluated and refined the proposed MMIC Framework 
using expert opinion in a two-round web-based modified 
Delphi process. The Delphi method is a consensus-building 



technique that solicits the opinions of content experts in a 
given field through the use of a series of questionnaires 
combined with the provision of feedback [35]. Using the 
participant list (n = 344) from the Health Council of 
Canada's National Symposium on Integrated Care, we 
selected and invited a random sample of 172 individuals 
to participate in our study (50% of the population). The 
Symposium, held on 10 October 2012, brought together 
policymakers, planners, managers, care providers, edu- 
cators, researchers and patient advocates from across 
Canada to share and promote the spread of innovative 
practices in integrated care. 

Following the Symposium, experts were invited via 
email to participate in the study. They were asked to com- 
mit to completing two questionnaires (-30 minutes each) 
and to have access to email or the Web for receiving and 
returning questionnaires. Completing the questionnaires 
implied consent to participate. Respondents were assured 
confidentiality, but not anonymity. Coffee cards ($10 each) 
were offered as an incentive for participating in both 
rounds of the study. For each round, two reminders were 
sent to non-respondents via email. The study received 
ethics approval from the Office of Research Ethics at the 
University of Toronto (protocol #28076). 

The questionnaire was developed and designed using 
Dillman s [36] criteria for question and questionnaire de- 
sign with Survey Monkey as the platform. Prior to ad- 
ministration, the questionnaire was pre-tested with eight 
volunteers (two care providers, one health services re- 
searcher, and five managers from a range of healthcare 
organizations) for clarity, and to anticipate the average 
completion time. All volunteers had knowledge of and/or 
experience in integration initiatives. As a result of the pre- 
test, modifications were made to the questionnaire s length, 
instructions, and lay-out. The pre-test also highlighted dif- 
ferences in how "integration" and "integrated care" were 
defined and perceived. All the volunteers preferred "mental 
models of integration" over "mental models of integrated 
care" because the latter focuses exclusively on direct pa- 
tient care to the perceived exclusion of other forms of inte- 
gration, such as organizational and functional integration. 
We changed the terminology accordingly. We also made 
minor modifications to two of three mental model cat- 
egories based on the pre-test: "integration-task mental 
model" and "integration-role mental model" were re- 
placed with "strategy mental model" and "roles mental 
model" respectively. 

The first round of the web-based questionnaire took 
place between 22 October and 12 November 2012 and 
consisted of 46 items (Additional file 1). The question- 
naire provided a two-page overview of the proposed 
framework, followed by questions which asked respon- 
dents to rate the clarity, comprehensiveness, usefulness, 
and importance of the concepts in the framework, and the 
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framework itself, using seven-point ordinal scales. Re- 
spondent demographics were also collected and spaces for 
comments were provided throughout the questionnaire. 

The second questionnaire, sent only to those who 
responded to the first round, took place between 4 
December and 20 December 2012 and consisted of 33 



items (Additional file 2). The questionnaire did not 
include questions on respondent demographics and 
questions on importance were not repeated due to high 
ratings from round one. The general format and remaining 
questions were the same as in the first questionnaire with 
three additions: (a) a summary of the feedback from round 



Table 1 Personal characteristics of respondents 



Characteristic 


Category 


Round 1 (n = 90) 


Round II (n = 68) 


Gender 


Female 


74.4% 


72.1% 




Male 


25.6% 


27.9% 


Age 


20-30 


13.3% 


14.7% 




31-45 


26.7% 


26.5% 




46-60 


45.6% 


41.2% 




61 + 


14.4% 


1 7.6% 


Region 


Central Provinces 


76.7% (73% ON) 


83.8% (79% ON) 




Western Provinces 


13.3% 


8.8% 




Atlantic Provinces 


6.7% 


3.0% 




Prairie Provinces 


3.3% 


4.4% 


Current Role 1 


Manager/Administrator 


56.7% 


52.9% 




Policymaker 


1 7.8% 


13.2% 




Researcher/Academic 


14.4% 


14.7% 




Consultant 


14.4% 


16.2% 




Clinician/Care Provider 


12.2% 


13.2% 




Patient/Caregiver Advocate 


6.6% 


5.9% 




Educator 


5.6% 


4.4% 


Current Workplace 1 


Coordinating/Advisory Body 


19.0% 


22.1% 




Hospital 


19.0% 


14.7% 




Home & Community Care 


16.7% 


16.2% 




Ministry/Government 


14.4% 


11.8% 




RHA/DHA/LHIN 2 


12.4% 


1 1 .8% 




Primary Care 


11.1% 


14.7% 




University or Research Institute 


11.1% 


1 1 .8% 




Professional Association/College 


9.0% 


8.8% 




Long-Term Care 


2.2% 


2.9% 


Level of Integration 3 


Micro 


23.3% 


23.5% 


(experience) 


Meso 


47.8% 


47.1% 




Macro 


28.9% 


29.4% 


Integration Activities 1 


Planning 


81.1% 


73.5% 


(experience) 


Implementation 


64.4% 


57.4% 




Management 


57.8% 


52.9% 




Evaluation 


55.6% 


52.9% 




Patient Care 


35.6% 


30.9% 




Policy-Making 


31.1% 


26.5% 




Research 


29.0% 


23.5% 




Patient/Caregiver Advocacy 


14.4% 


1 1 .8% 




No Direct Experience 


3.3% 


4.4% 



Respondents could select multiple options. 

2 RHA/DHA: Regional/District Health Authority; LHIN: Local Health Integration Network. 

3 Macro: integration for entire communities; Meso: integration for specific patient or caregiver populations; Micro: integration for individual patients and 
their caregivers. 
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one, including response rate, participant demographics, 
and major findings, (b) an enhanced introductory explan- 
ation of the framework, its background, and its potential 
applications, and (c) quantitative and qualitative feedback 
from the first questionnaire for each question. The feed- 
back included an outline and explanation of the modifica- 
tions made and not made to the framework based on the 
first round results. The second questionnaire was pre- 
tested for clarity and ease of completion prior to adminis- 
tration with a sub-set of four volunteers from the original 
group of eight from round one. No changes were made to 
the questionnaire as a result of the pre-test. 

The results for both questionnaires were tabulated 
using quantitative and qualitative analyses. Descriptive 
statistics were used to describe the structured responses 
as a whole, while the open comments were coded and 
categorized using thematic analysis to identify common 
themes and key issues. The Kruskall-Wallis test was used 
to examine cross-group agreement by level of integration 
experience, current workplace, and current role. In the 
questionnaire, respondents had the option of selecting 
multiple roles and workplaces; less than 5% (n = 4) and 
15% (n = 12) chose to do so respectively. For the purpose 
of analysis, however, each respondent was assigned only 
one role and one workplace based on the information they 
provided in the questionnaire (i.e., primary affiliation). 

Results 

Round I questionnaire 

In the first round of the Delphi process, a total of 90 in- 
dividuals responded from nine provinces across Canada 
(52% response rate). As outlined in Table 1, respondents 
represented a wide range of roles and organizations in 
the healthcare system, and 96.7% had direct experience 
in integration activities. Over 250 comments were made 
in total. The responses to the questions are shown in 
Figures 1 and 2, comparing the results of both rounds of 
the study. Participants were asked to rate the importance 
of having shared knowledge/beliefs for each of the men- 
tal model contents only once - either in the first round 
for the original set of contents or in the second round 



for newly added contents. In both rounds, the average 
ratings for importance were high: 75% and 84% respect- 
ively (percent strongly agree, defined as a score of 6 or 7 
on a 7-point scale); hence these results are not included 
in the Figures. The ratings for the clarity, comprehen- 
siveness and usefulness of the terms and definitions were 
lower than those for importance with, on average, only 
half of respondents providing a high rating (defined as a 
score of 6 or 7 on a 7-point scale); the results were simi- 
lar for respondent views of how useful the framework is 
for interpreting, planning, implementing, managing and 
evaluating integration initiatives. 

The main themes elicited from the review of open 
comments and the changes made in response to them 
are summarized in Table 2. Based on the quantitative 
and qualitative analysis, key findings included the need 
to: explain the framework s purpose and potential appli- 
cations, clarify some terms and definitions, modify the 
structure of the framework, and include additional con- 
tent. The most important changes involved modifying 
the title of the framework from "Mental Models of Inte- 
gration" to "Integration Mindsets" to enhance clarity and 
understanding, and the inclusion of eight new content 
areas in the beliefs/perceptions components of the 
framework to address gaps identified by the respondents. 
These additional content areas were identified using re- 
spondent comments (Table 3) and are supported by lit- 
erature on health systems integration [e.g., 7,9,11]. 

Round II questionnaire 

In the second round, 68 individuals responded (75.6% 
response rate). Over 95 comments were provided. In gen- 
eral, the ratings and comments were less critical than 
those provided in round one. Less than 3% of respondents 
gave a low rating (between 1 and 3 on a 7-point scale) for 
any of the survey items. The overwhelming majority of re- 
sponses (85-96%) fell between 5 and 7 on a 7-point scale 
compared to only 63-82% in the first round, representing 
an average increase of 16%. The ratings increased sub- 
stantially for the practical applications of the framework 
as a whole and for the Strategy Mental Model contents 




■ 1st Round - all respondents 



Comprehensive 
Attributes of the Strategy Mental Model 



Clear Comprehensive Useful 

Attributes of tho Relationships Montal Model 



Figure 1 Delphi results: Attributes of the mental models. Are the definition, description and contents of the Strategy/Relationships Mental 
Model clear, comprehensive, and useful? Average percent strongly agree (rating of 6 or 7 on a 7-point scale). 
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100 




1 1st Round - all respondents 



1 1st Round - second round 



respondents 
2nd Round 



Application 

Figure 2 Delphi results: Practical applications of the framework. Is the framework, as a whole, useful for each of the following applications? 
Average percent strongly agree (rating of 6 or 7 on a 7-point scale). 



(Figures 1 and 2 respectively). For the Roles Mental 
Model (later renamed the Relationships Mental Model), 
the ratings only increased for comprehensiveness while 
views on clarity remained the same and views on useful- 
ness decreased (Figure 1). This result suggests that there 
may be a trade-off between comprehensiveness and com- 
plexity, and the perceived usefulness of the contents. 

The main themes elicited from the review of open 
comments and the changes made in response to them 
are summarized in Table 2. The quantitative and qualita- 
tive analysis identified the need to further clarify some 
terms and definitions as well as to reiterate the scope of 
the framework. Although ratings for comprehensiveness 
increased by about 20% in round two for both the Strat- 
egy and Roles Mental Model, upon analysis of respond- 
ent comments, it was apparent that most respondents 
were judging comprehensiveness based on the frame- 
works inclusion of influencing factors. Several respon- 
dents suggested that leadership, organizational culture, 
and policy be included in the framework. Although these 
are important factors that influence the implementation 
and success of integration initiatives, they describe the 
context for integration and not knowledge and beliefs 
regarding integration strategies. 

Cross-group agreement by level of integration 
experience, role, and workplace 

Analysis of cross-group agreement revealed some statisti- 
cally significant group-level differences in responses, pri- 
marily in ratings of usefulness of the concepts and 
framework (Additional file 3). Among the three respondent 
characteristics tested - level of integration experience, 
current role, and current workplace - only level of integra- 
tion experience had a consistent relationship with group- 
level responses across both rounds of the study. In the first 



round, respondents with micro-level integration experience 
provided higher ratings than those with meso- and macro- 
level experience in select areas, primarily related to useful- 
ness. In the second round, however, a significant difference 
between groups was detected for only one survey item 
which examined the usefulness of the framework for man- 
aging integration. Those with meso-level integration ex- 
perience provided a lower mean rating (5.6 on a 7-point 
scale) for this question than those with micro- and macro- 
level experience (6.1 on a 7-point scale). 

In regards to current role, respondents in a management, 
administration or consultant role initially provided higher 
ratings in comparison with other groups, particularly re- 
searchers, for the usefulness of the framework for inter- 
preting, planning, implementing, managing, and evaluating 
integration. However, in the second round there were no 
significant differences identified in ratings across the four 
role types, which may suggest an increase in cross-group 
agreement. While ratings did not vary by respondents' 
workplace in round one, they did in round two in relation 
to two survey items. For these two survey items, which 
examine usefulness of the "Roles Mental Model" concept 
and usefulness of the framework for interpreting integra- 
tion experiences, ratings were generally lower among those 
working for a coordinating or advisory body, or for the 
government, than those working in other settings. This 
may be because individuals working in these settings are 
farther removed from direct patient care. For example, 
most of the coordinating and advisory bodies represented 
in the sample are focused on health system performance 
measurement and reporting. 

In summary, the results suggest that respondents typ- 
ically agreed on the clarity, comprehensiveness, and im- 
portance of the proposed concepts and framework; the 
key point of contention was in regards to how useful the 
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Table 2 Results from thematic analysis of open comments 



Main themes 



Modifications made 



Comments 



Round One 

Clarify the: 

• Purpose of the framework 

• Structure of the framework, 
particularly the section on 
beliefs/perceptions 

• Practical applications of the 
framework 

• Terms and definitions in 
some parts of the 
framework, including the 
terms "mental model" and 
"shared" 

• Patient/caregiver 
perspective and role 

Include additional content, 
particularly in the area of 
beliefs/perceptions 



Round One 

Provided a three-page overview to explain: 

• The value of a cognitive perspective on integration 

• The broad meaning of the term "shared" 

• What the framework aims, and does not aim, 
to do 

• How the framework may be used, including 
examples 

Replaced the term "Mental Models of Integration" 
with "Integration Mindset" 

Removed "belief/perceptions" as a separate type of 
mental model and added it as a component to the 
two remaining mental model types 

Developed additional content to include in the 
framework, primarily on beliefs/perceptions: 

• Strategy: "evaluation" added as a knowledge 
component and four beliefs/perceptions added 

• Roles: four beliefs/perceptions added 

Edited several terms and definitions in the 
framework for clarity and comprehensiveness. For 
example: 



Round One 

We retained the term "mental model" for two 
reasons: (a) no other term conveys the intended 
meaning and (b) the term maintains the link 
between this work and the literature on mental 
models. 

In the original framework, the beliefs/perceptions 
mental model consisted of the same contents as in 
the knowledge category, but with a focus on "what 
should be" or "what is perceived or expected". For 
example, "clients" can be viewed in terms of which 
populations are being targeted for integrated care 
(knowledge of the integration strategy) versus 
which populations should be targeted (personal 
belief/perception). Repetition of the content areas 
created confusion and the contents failed to 
capture important beliefs/perceptions. Respondents 
also noted that their way of thinking about 
integration was complex and dynamic, 
incorporating knowledge, beliefs and perceptions; 
they therefore did not view beliefs/perceptions as a 
separate mental model. 

Additional framework contents identified through 
respondent comments and literature. 



Round Two 

Reduce the number of 
content areas to improve 
clarity and reduce overlap 

Further clarify some terms 
and definitions in the 
framework 

Further explain why factors 
such as culture and 
leadership are not included 
in the framework 



• Strategy: "services to be integrated" changed to 
"targets"; "external customers changed to "clients"; 
and "processes" changed to "methods" 

• Strategy: the definition for knowledge of clients 
was changed from "patients/caregivers who will 
benefit from integration" to "characteristics and 
needs of the populations, patients and/or 
caregivers who will benefit from integration, and 
the nature of that benefit" 

• Roles: "knowledge and skills" changed to 
"competencies"; "role recognition" changed to 
"accountabilities"; and "interaction mechanisms" 
changed to "communication". Minor modifications 
were made to the definitions for these terms as 
well (i.e. "role recognition: the purpose and 
responsibilities of each professional, organization 
and client" was replaced with "accountabilities: 
the activities and results that team members are 
individually or jointly responsible for"). 

Round Two 

Replaced the term "Roles Mental Model" with 
"Relationships Mental Model" 

Applied minor edits to some terms and definitions. 
For example: 

• Strategy: "aptitude for change" changed to 
"readiness for change" 

• Relationships: we originally described the content 
for "each participating professional" and "each 
unit", the latter defined as a program, department 
or organization. To improve clarity we now 
describe each content area using inter- 
professional and inter-organizational "team 
members" as the referent point. 



Round Two 

Reducing the number of content areas may be 
premature without further research. 

Several respondents noted that the term "role" 
focused attention on individual professionals and 
emphasized boundaries. 

The next stage of this research aims to develop a 
measurement tool for capturing and comparing 
Integration Mindsets as well as practitioner tools. 
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Table 2 Results from thematic analysis of open comments (Continued) 

Develop an assessment tool • Relationships: "recognition of shared responsibility: 

or checklist to facilitate willingness to go beyond what one is obliged to 

application do to support or contribute to the integration 

process" was replaced with "recognition of shared 
responsibility: a willingness to share the burden of 
work and act as a team to contribute to the 
integration process and/or to the delivery of 
integrated care" 

Updated the three-page overview with an 
explanation of the range of contextual factors 
that influence (but are not inherently a part of) 
Integration Mindsets 



concepts and framework are in practice and in what 
ways. Such differences are expected given the diversity 
of integration foci and methods, and the variety of pro- 
fessional groups and organizations involved in integra- 
tion. Respondents who were "closer" to patients, such as 
managers and care providers working in healthcare set- 
tings, generally provided higher ratings than researchers, 
policymakers, or those working for advisory bodies. 
However, there were fewer differences detected in round 
two compared with round one, which may indicate in- 
creased cross-group agreement. 

Overview of the integration mindsets framework 

The final version of the Integration Mindsets Framework 
is outlined in Table 4. An "Integration Mindset" refers to 
an individuals way of thinking about integration that is 
based on knowledge and beliefs regarding the strategy for 
achieving integration (i.e., Strategy Mental Model) and the 
roles and relationships of those involved in the integration 
process (i.e., Relationships Mental Model). The contents 
of these two mental models, defined in Table 5, represent 
areas where a lack of shared knowledge and shared beliefs 
may negatively impact integration efforts. The framework 
is not intended to capture the process of integration or to 
reflect all of the factors that influence integration. The 
framework is also not designed as a prescriptive tool to 
"impose" strategies and "assign" roles. Rather, the frame- 
work is intended to aid discussion and measurement. The 
aims are to identify which knowledge and beliefs influence 
integration and to enhance understanding of the multiple 
conceptualizations of integration so that differences can 
be identified, unpacked, and explored; the desired out- 
come of this process is a synthesis of perspectives that is 
qualitatively better than any of the individuals' perspec- 
tives. The framework is intentionally broad to permit its 
use for inter-organizational and inter-professional teams 
at macro, meso, or micro levels. The framework may 
also be adapted for use at different stages of integration. 

Discussion 

The Integration Mindsets Framework aims to identify 
the cognitive contributors to integration success or 



failure. The framework outlines important knowledge 
and beliefs whose convergence or divergence across 
stakeholder groups may influence inter-professional and 
inter-organizational relations. Although SMMs are only 
one of many factors that influence integration, they may 
help to explain variations in performance, and may be 
leveraged to accelerate change in tandem with modifica- 
tions to policy and legislation, organizational structures 
and context, and administrative and care processes. 

A cognitive perspective on integration enhances our 
understanding of normative integration and facilitates its 
measurement, in part by complementing a cultural lens. 
Frameworks and methods used to measure "integrated 
system culture" provide limited, high-level insights [37] 
that are often not actionable, which may explain why 
integration evaluation methods rarely measure the cul- 
tural aspects of integrating [38]. The Integration Mind- 
sets Framework allows us to explore and potentially 
measure the micro-foundations of culture with a focus 
on knowledge and beliefs specific to integration as op- 
posed to general cultural attributes. Knowledge is of par- 
ticular importance for two reasons: (1) the absence of 
knowledge in frameworks of organizational culture, and 
(2) knowledge of a change (i.e., what the change is, how 
it will impact the organization, how it will be imple- 
mented, etc.) has been linked to less resistance to change 
[39]. Questionnaires or interview guides may be devel- 
oped to collect data on each of the content areas listed in 
Table 4 from various stakeholders involved in an integra- 
tion activity, including managers and clinicians from orga- 
nizations across the continuum of care such as hospitals, 
long-term care homes and community-based agencies. 
This type of data will complement the high-level informa- 
tion captured by cultural tools on an organizations orien- 
tation in regards to people, innovation, control and 
outcomes [10], and provide a more complete understand- 
ing of normative integration in a given context and at a 
given point in time. 

The two-round Delphi process allowed the original 
framework by Evans and Baker [20] to be tested against 
expert opinion. The study design created an opportunity 
for structured dialogue on integration among a varied 
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Table 3 Sample respondent comments used to identify beliefs/perceptions content 



Content 



Sample respondent comments 



Strategy Mental Model 

Consequences of integrating 



Appropriateness of selected 
strategy 



Integrity of decision-making 
processes 

Readiness for change 



Relationships Mental Model 

Appropriateness of role structure 



Identification with the integration 
initiative 



Recognition of shared 
responsibility 



Importance of client involvement 



"To be successful each participant must see benefit for him/her as well as the collective" 

"Without willingness to accept and acknowledge the value of integration, it will be difficult to make progress" 

"Need to include the concept of agreement with the goals, long-term vision and methods" 

"Although there may be shared knowledge unless there is acceptance and commitment there will be 
challenges" 

"Decisions [must] have clear rationale that can be publicly defended" 
"Important to have all views represented at the decision-making table" 

"Add a concept about willingness to put energy into finding out, testing hypotheses, innovation" 

"Include individual's belief about their perceived freedom or ability to make change or execute integration 
activities" 

"It goes beyond knowledge to understanding, appreciation and agreement on the roles" 
"Knowledge without buy-in is not sufficient" 

"Each participant must have enthusiasm and investment in the initiative and must think beyond their current 
boundaries" 

"Crucial to know why they are there: voluntarily or a directed (forced) integration?" 

"If the involved parties don't recognize the need for specific organization engagement, the overall integration 
activity may not gain traction" 

"More emphasis on interdependence, being more collaborative, recognizing that each brings expertise" 

"The key issue is the integration of patient/family into interprofessional teamwork and this is dependent on 
changing current attitudes about practice and patient involvement" 

"Most important players in planning and implementing integration are the persons and their families" 



group of healthcare stakeholders, and between investiga- 
tors and participants. The respondents represented diverse 
policy contexts, professional roles, and organization types, 
thereby supporting the validity and generalizability of the 
concept of "Integration Mindsets" and the accompanying 
framework. Respondents asked questions, offered sugges- 
tions, and shared examples from their professional experi- 
ences; this rich feedback helped to clarify and further 
develop the concepts, structure and practical value of the 
framework while retaining and validating the underlying 
theoretical foundations. After the second round, there was 



strong interest among participants in the development of 
assessment or discussion tools for use in the field as well 
as vignettes or cases demonstrating use of the framework. 

Many of the limitations associated with the Delphi 
method relate to small sample size and the identifica- 
tion, selection and commitment of the expert panel 
members [35]. In this study, we used a random sample 
of experts who attended a national symposium on inte- 
grated care, thereby reducing selection bias. Our sample 
was relatively large (n = 90) and response rates relatively 
high (52% and 76% respectively) in comparison to most 



Table 4 Integration mindsets framework 



Mental Model Type 


Definition 


Knowledge (information 


Beliefs/Perceptions (opinions or 






& awareness) 


internal feelings) 


Strategy Mental 


A conceptualization of what is being 


• Targets 


• Consequences of integrating 


Model 


integrated and how, why and for 


• External clients 


• Appropriateness of selected strategy 




whom it is being integrated 


• Goals 


• Integrity of decision-making processes 






• Long-term vision 


• Readiness for change 






• Methods 








• Evaluation 




Relationships Mental 


A conceptualization of the organizations, groups, 


• Competencies 


• Appropriateness of role structure 


Model 


and individuals (including one's self) involved in 


• Contributions 


• Identification with the integration initiative 




integration and how they are connected 


• Accountabilities 


• Importance of client involvement 






• Interdependencies 


• Recognition of shared responsibility 






• Communication 
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Table 5 Definitions of concepts in the integration mindsets framework 



Term 



Definition 



Strategy Mental Model - Knowledge Content 



Targets 

External clients 
Goals 

Long-term vision 
Methods 

Evaluation 



functions, services, organizations and/or systems identified for integration 

characteristics and needs of the populations, patients and/or caregivers who will benefit from integration, 
and the nature of that benefit 

primary aims of integration, which may be related to costs, efficiency, productivity, quality of care, patient 
safety and/or patient outcomes 

how the services, programs or functions, organizations and/or systems will "look" or operate when fully integrated 

approaches and enablers for achieving integration - which may be clinical, technological, patient or 
caregiver-centered, administrative, financial, organizational, governance and/or policy-related - and timeline for 
implementation 

key performance dimensions and indicators for assessment of the integration initiative 



Strategy Mental Model - Beliefs\Perceptions Content 



Consequences of integrating 

Appropriateness of selected 
strategy 

Integrity of decision-making 
processes 

Readiness for change 
Relationships Mental Model - Knowledge Content 



the expected outcomes (positive and/or negative) of integration for one's self, for other participating individuals 
and organizations, for external clients, and for the healthcare system 

the extent of agreement with the selected targets, clients, goals, long-term vision, methods and evaluation 
approach for an integration initiative 

the extent to which decisions regarding integration are made in a manner that is equitable and transparent 
the ability and willingness to implement the desired integration initiative 



Competencies 

Contributions 

Accountabilities 

Interdependencies 

Communication 



the knowledge and skill sets of each team member 

how each team member contributes to patient health and well-being 

the activities and results that team members are individually or jointly responsible for 

how and to what extent the work of each team member depends on or is influenced by another 

sources of information and how information flows between team members, including frequency and 
methods for contact 



Relationships Mental Model - Beliefs\Perceptions Content 

Appropriateness of role structure the extent of agreement with the content and distribution of roles, including relative accountabilities and 

communication methods 

Identification with the integration the extent of self-association with the integration initiative (i.e. the team, partnership, network, etc.) in 
initiative addition to one's professional group and organization 

Recognition of a willingness to share the burden of work and act as a team to contribute to the integration process and/or 

shared responsibility to the delivery of integrated care 

Importance of client involvement the extent to which the involvement of patients and their caregivers is considered necessary and beneficial 

to integration efforts 

^he term "team member" is used broadly to refer to individuals, organizations, and patients/caregivers participating in the integration initiative; the composition 
of the team will depend on the nature and level of the integration activity. These teams typically span professional and organizational boundaries, may be 
focused on governance, management or patient care, and may be formal or informal and ad hoc/intermittent or fixed. 



Delphi and survey studies respectively [35,36]. Although 
respondents were overwhelmingly female (74%), from 
the province of Ontario (73%), and in a management 
position (57%), these distributions reflect the broader 
population of symposium attendees. While the diversity 
of the respondents and organizations represented sup- 
port generalizability of the framework, further research 
is required to determine generalizability beyond the 
Canadian context. In addition, patient and caregiver per- 
spectives were not captured because the framework and 
method of study would require modification to 



appropriately reflect and elicit their views; however, the 
framework does conceptualize patients and caregivers as 
valued members of the healthcare team, and patient ad- 
vocates did participate in the study (n = 6). Finally, al- 
though the Kruskall-Wallis test is robust in detecting 
differences across groups with small sample sizes (n > 5), 
the sub-group sample sizes by respondent level of inte- 
gration, workplace and role were unequal with relatively 
large ranges (e.g., fifty-three managers versus eleven re- 
searchers). The results must therefore be interpreted 
with caution. 
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Conclusion 

Given the cultural and cognitive challenges to integra- 
tion [6-9,40,41], there is an increasing need for research 
and practical tools that can enhance our understanding 
and ability to measure and influence normative integra- 
tion. The Integration Mindsets Framework offers a start- 
ing point for doing so by drawing our attention to how 
various stakeholders involved in a specific integration 
initiative think about and conceptualize integration. The 
cognitive and cross-boundary (i.e., inter-organizational 
and inter-professional) focus of the framework supports 
recommendations from the change management literature 
to engage individuals and groups at all levels in leading 
and participating in change efforts, to pay attention to 
change recipients' emotional, cognitive, and behavioral re- 
actions to change, and to address divergent visions and 
goals or mis-alignments in the knowledge and beliefs 
needed for the change [42-44]. 

We can use the Integration Mindsets Framework to 
explore: How do mindsets evolve (or not) with imple- 
mentation? How do views differ among leaders and staff, 
providers and managers, and providers and patients/ 
caregivers? How similar are mindsets among different 
organizations involved in integration? These questions 
may be explored through discussions among team mem- 
bers or partnering organizations using the framework as a 
guide, or through formal measurement once a measure- 
ment tool has been developed for capturing and compar- 
ing integration mindsets. Below we offer more specific 
examples of potential applications of the framework. 

• Interpretation: Current or past integration efforts 
may be (re-) interpreted using the framework. For 
example, despite careful redesign of structures and 
processes, and a favourable environment, some 
integration efforts still fail to meet objectives. 
Differences in integration mindsets may help partly 
explain such cases. 

• Planning: The framework can be used to direct and 
focus early discussions and planning efforts among 
team members or partnering organizations, and to 
assess system or organizational readiness for 
integration. 

• Implementation: The framework draws our 
attention to important knowledge content, some of 
which may be co-created or clarified, recorded, 
and formally agreed to during the integration 
implementation stage. 

• Management: Awareness of the extent to which 
integration mindsets are shared and where 
similarities and differences lie can help guide 
change management interventions. A lack of 
cross-understanding (possessing an accurate 
understanding of the mental models of others) 



negatively impacts collective learning and 
performance [45]. Managers and leaders can use 
information on integration mindsets to identify 
pockets of resistance to change or to leverage the 
support of champions, to re-frame the initiative, as 
well as to develop interventions which align with 
the identified problem [46,47] . With regards to the 
latter, for example, a lack of shared knowledge may 
be addressed through training and education, 
whereas a lack of shared beliefs may require more 
extensive and potentially long-term dialogue and 
negotiation in addition to changes to structures and 
incentives [20]. 
• Evaluation: The extent to which integration 

mindsets are shared may help assess the success and 
sustainability of an integration activity. As 
relationships develop and new work practices 
become embedded over time, actors retrospectively 
make sense of what they did together and 
knowledge and beliefs/perceptions become more 
congruent [48,49]. "Shared integration mindsets" 
may be used as one indicator, among many, of a 
successful and sustainable integration activity. 

In addition to improving the Integration Mindsets 
Framework in preparation for empirical research and 
practical use, this study also establishes a common lan- 
guage for discourse on cognition in health systems inte- 
gration and encourages interdisciplinary and cross-level 
theorizing. Further research is needed to determine the 
contribution of the Integration Mindsets Framework to 
research and practice. In particular, future studies should 
examine the frameworks relevance in various countries 
and health systems, and develop and test measurement 
tools for capturing and comparing "integration mindsets" 
with attention to psychometric properties. An inter- 
national Delphi panel is an appropriate next step to exam- 
ine the frameworks generalizability beyond the Canadian 
context and to further validate the framework prior to its 
application. Incorporating patient views is also fundamen- 
tal to the validation process. Complex patients accus- 
tomed to interacting with multiple care providers may be 
briefed on the structures and processes associated with a 
particular integrated care model, and asked what they 
think their providers need to agree on - and what shared 
knowledge and beliefs they require - to work together ef- 
fectively under that model. This open-ended discussion 
may be followed by a more focused discussion of the con- 
tents of the framework and the extent to which the con- 
tents resonate with patients. Future research will also help 
clarify the nature of the frameworks contents and their 
relative importance to collaboration. For example, with 
regards to the "evaluation" component of the Strategy 
Mental Model, is having a shared awareness of what the 
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metrics are important in and of itself? Or is having a 
shared belief in the validity of the metrics and individual 
or collective ability to influence the metrics important? In 
addition, how might the influence of convergent and di- 
vergent integration mindsets differ based on the type and 
level of integration, the degree of similarity or dissimilar- 
ity, the number and diversity of actors involved, and the 
context? In summary, a better understanding of the evo- 
lution and interplay of meanings, interpretations and 
knowledge about integration across inter-professional 
and inter-organizational boundaries may help to accel- 
erate progress towards integrated care. 
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